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ABSTRACT 

The impact of an outreach program modeled after the 
^jrkeley Bay Area Writing Project was studied. Both projects 
emphasize the role of expert teachers as teacher trainers for the 
purpose of improving student writing. The University of California at 
Los Angeles (UCLA) program targets teachers of traditionally 
underserved students. Data sources included writing samples from 274 
junior and senior high school students and questionnaire responses 
from 333 high school seniors in project schools. Results clearly 
indicate that students in the classes of teachers who are 
writing-project fellows (graduates of the summer training institutes) 
write significantly bettor than students not in program classes. 
Results also indicate that the quality of student writing is 
positively related to the number of project fellows they experience 
as Engl ish teachers. Other variables are even more powerful 
predictors, such as grade point average (GPA) , school socioeconomic 
index, and school level. Findings related to the effects of the 
Writing Project on student GPA, pos t s econdary education plans, 
college entrance test scores, and attitudes are less clear. 
Implications for future studies are discussed. (SLD) 
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SUMMARY 



Tite ir metus for this study was a desire on the part of the University of California, 
Los Angeles's (UCLA's) Graduate School of Education to evaluate the impact of an 
outreach pro* am modeled after the Berkeley Bay Area Writing Project and parented by the 
National and Talifornia Writing Projects. These projects emphasize the role of expert 
teachers as teacher trainers for purposes of improving student writing. UCLA's program, 
administered through the Center for Academic Interinstitutional Programs (CAIP), 
particularly targets teachers of traditionally underserved students. While the CAIP staff 
have been regular in their collection of survey data, data collection has been unsystematic 
and evaluation efforts have been aimed at teacher participants rather than students. In 
short, evidence of program impact on students has been lacking. 

In the present study, data sources included student writing samples and student and 
teacher questionnaires. Writing samples were collected from a culturally diverse population 
of 274 junior and senior high school students in schools targeted by the UCLA Writing 
Project. Questionnaire data were obtained from 333 high school seniors in schools served 
by the Project. 

Results clearly indicate that students in the classes of teachers who are Writing 
Project Fellows (i.e., graduates of intensive summer institutes conducted by CAIP staff) 
write significantly better than students who are not. Results also indicate that the quality of 
students' writing is positively related to the number of Writing Project. Fellows they 
experience as English teachers. While level of teacher involvement in the Writing Project 
and level of student exposure to Writing Project Fellows are significant predictors of 
writing quality, it is not surprising that other variables provided more powerful predictors: 
self-report grade point averages (CPAs), school socioeconomic index, and school level 
(i.e., junior high versus senior high school). 

Findings related to the effects of the Writing Project on students' CPAs, 
postsccondary education plans, college entrance test scores, and attitudes toward writing 
are less clear. Findings suggest that CPAs and ethnicity are the most influential variables 
in students' decisions about whether and where they will continue their education after high 
school. In relative terms, UCLA, fci instance, attracts more Asian minority students than 
Black or Hispanic minority students. 

Aspects of the evaluation approach described in this report that should be retained in 
future studies include (a) the systematic collection and analysis of student products such as 
writing samples and (b) design consideration of moderator variables such as language 
spoken at home, socioeconomic status (SES), school context, and intensity and validity of 
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the intervention or program under evaluation. The success of future efforts to assess the 
impact of university outreach programs on students' decisions to enter university systems 
and these students' university success requires careful attention and commitment from 
central administration to developing mechanisms for (a) tracking cohorts of university 
students backward to determine the nature and extent of their involvement in targeted 
outreach programs and (b) tracking high school students forward longitudinally to 
determine how various levels of involvement in outreach programs affect their future 
educational planning, decision-making, and success. 
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Impact Evaluation of a University-Based Outreach Program Modeled After 
the Berkeley Bay Area Writing Project 

The impetus for this study was a desire on the part of the University of California, 
Los Angeles's (UCLA's) Graduate School of Education to evaluate the impact of an 
outreach program modeled after the Berkeley Bay Area Writing Project and parentcd by the 
National and California Writing Projects. These projects emphasize the role of expert 
teachers as teacher trainers for purposes of improving student writing. UCLA's program, 
administered through the Center for Academic Interinstitutional Programs (CAIP), 
particularly targets teachers of traditionally underserved students. While the CAIP staff 
have been regular in their collection of survey data, data collection has been unsystematic 
and evaluation efforts have been aimed at teacher participants rather than students. 

The focus of this evaluation study was on student outcomes. A representative 
sample of the population served by CAIP was selected for study; under-represented 
students were included. Two major questions were addressed by the study: 

Does student writing differ qualitatively depending on teachers' levels of exposure 
to and participation in the UCLA Writing Project, and if so, what are the 
differences? 

How do self-report grade point averages (GPAs), Scholastic Aptitude Test (SAT)- 
Verbal scores, postsecondary plans, and attitudes differ among high school seniors 
with varying levels of exposure to UCLA Writing Project teachers? 

Method 

The following data were collected: (a) writing samples and questionnaires from 
274 students nrolled in Los Angeles area junior and senior high schools as well as 
questionnaires completed by these students' current teachers and (b) questionnaires only 
from 333 high school seniors and their current teachers. 
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Writing Quality 

The students of four junior high school and six high school teachers participated in 
this aspect of the study. The junior high schools were located in Glendale, inner-city Los 
Angeles, and Mission Viejo, California. The high schools were located in Beverly Hills, 
inner-city Los Angeles, and Reseda, California. 

While a sampling goal was to achieve a balance between students of both UCLA * 
Writing Project Fellows and teachers with no Project exposure, coding and analysis 
procedures took other levels of Project participation or exposure into account . In fact, 
however, the analyses demonstrated that all teachers participating in the study either were 
Writing Project Fellows or had never received information or other exposure to the Project. 
(Writing Project Fellows are graduates of intensive summer institutes conducted by CAIP 
staff over several weeks usually consisting of six-hour days. These graduates 
subsequently serve as CAIP consultants and as Writing Project teacher trainers.) 

Of the two junior high school teachers in inner city schools, one is a UCLA Writing 
Project Fellow and the other reported no exposure to the Project. Of the two junior high 
school teachers from more suburban areas, one is a UCLA Writing Project Fellow and the 
other reported no exposure to the Project. 

Two of the high school teachers who participated in die study work with bilingual 
students enrolled in English as a Second Language (ESL) classes. One of these teachers is 
a UCLA Writing Project Fellow; the other reported no exposure to the Project. Another 
two of the high school teachers are from inner city schools; one is a UCLA Writing Project 
Fellow and the other reported no exposure to the Project. The two remaining high school 
teacher participants are from more suburban areas; one is a UCLA Writing Project Fellow 
and the other reported no exposure to the Project. 

Altogether, written essays and contextual information provided by the Student and 
Teacher Questionnaires (available upon request from the author) as well as public records 
were collected on 274 students. The contextual information included the extent to which 
the students' current and past three English teachers were exposed to or participated in the 
UCLA Writing Project, self-report cumulative GPAs, English grades for the past three 
years, grade level (i.e., junior high school versus, high school), school socioeconomic 
status (SES) as determined by the most recent California Achievement Program (CAP) 
designation, and school proportion of Limited English Proficient (LEP) students, also 
provided by the most recent CAP designations. 



ERLC 



6 



Writing Impact 
6 

In preparation for writing the essays, students read a passage by Richard Wright 
entitled "Hunger" (available upon request) and participated in a lesson designed by their 
individual teachers to make the passage accessible to them. Upon completion of the lesson, 
students were asked to write an essay in response to a standardized prompt. Copies of the 
writing prompt and teacher and student instructions are available upon request. Neither the 
students nor their teachers knew the prompt prior to its administration. All students were 
given one class period in which to write a response. Class periods were similar in length 
across classrooms. 

Results were analyzed in tv/o ways as detailed in the appendix: 

1 . They were scored holistically using a rubric modeled after that used by the 
California Assessment Program. 

2. They were scored holistically and analytically using the Content Assessment 
methodology developed by Eva Baker and colleagues (1991) at the National Center 
for Research on Evaluation, Standards, and Student Testing (CRESST). 

The Content Assessment analyses yield the following scores: 

General Impression-Content Quality (CICQ). The extent to which the student 
understands the content of the "Hunger" passage and integrates that understanding 
with personal experience in responding to the prompt. 

Principles/Concepts-Number (PN). A measure of the number of different 
passage-relevant concepts or principles that the student uses with comprehension. 

Prior Knowledge: Facts and Events (PK). A measure of the extent to which 
students incorporate relevant concrete information that is not mentioned in the 
passage into their essays. 

Proportion of Text Detail (TEXT). A measure of the amount of material from the 
passage text that is used in the essay. In the case of the "Hunger" essays, a 
relatively low proportion of TEXT was desired. 

• Misconceptions. A measure of the amount of incorrect information, or the number 
of misconceptions or misinterpretations in the essay. The "Hunger" essays did not 
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lend themselves to appropriate scoring for Misconceptions; hence, Misconception 
scores were not assigned to the essays. 

Interrater reliability for the two raters who did the California Assessment Program 
(CAP)-based holistic scoring was .62 based on 29 essays. Intrarater reliability was .82 for 
the rater who scored the essays that were not included in the reliability study. Intrarater 
reliability was based on the rescoring of 10 essays after two weeks. 

Interrater reliability for the two raters who did the Content Assessment scoring, 
based on 50 essays, was .88, .83, .88, and .84 for the GICQ, PN, PK, and TEXT scales, 
respectively. After a two week period, intrarater reliability for rater #1, based on 10 essays 
was .90 (GICQ), .87 (PN), .93 (PK), and .96 (TEXT). Over the same period and based 
on the same essays, intrarater reliability for rater #2 was .74 (GICQ), .88 (PN), .87 (PK), 
and .93 (TEXT). 

Self-Report Grades, Test Scores, Plans, and Attitudes 

Completed questionnaires (Senior English Student and Teacher Questionnaires; 
av^Iable from the author) were obtained from 333 high school seniors and their teachers. 
These students were enrolled in the classes of 15 teachers in schools representative of those 
targeted by CAIP. To determine the extent of students' exposure to Writing Project 
Fellows, questionnaire responses were checked against CAIP's database of Writing Project 
participants/Fellows. The purpose of the questionnaire data was to determine the 
relationships among the following: (a) the level of student exposure to Writing Project 
Fellows (ranging from one to four teachers); (b) the level of students' present and past 
three teachers' participation in the Writing Project (ranging from Fellow to no knowledge 
of the project); (c) students' postsecondary plans (ranging from working fulltime to some 
combination of school and work, to fulltime continuing education in a community college 
versus a branch of the California State College system versus a branch of the University of 
California system, particularly UCLA); (d) attitudes toward writing; (e) school 
socioeconomic code; (f) student ethnicity; (g) home language; (h) self-report, cumulative 
GPA; (i) average GPA in high school English classes; and (j) self-report score on the 
verbal portion of the SAT. 

The students who responded to the questionnaires were not the same students as 
those who wrote essays for the writing analyses aspect of the study. The schools of 
participating teachers and their students were located in Chatsworth, Hawthorne, inner-city 
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Los Angeles, Reseda, San Pedro, Santa Monica, Sylmar, and Torrance, California. Data 
on the ethnicity and primary language of the students are shown below. 



Ethnicity 

Asian 

Black 

Caucasian 

Hispanic 



n 

47 
45 
98 
120 



14.1 
13.5 
29.4 
36.0 



] an gnage Sp nkftn at Home 

English 

Spanish 

Other 

Bilingual 



II 

200 
56 
44 
30 



60.1 
16.8 
13.2 
9.0 



QuesUonnaires were administered by students' classroom teachers. Copies of the 
teacher instructions and student questionnaires are available upon request. 

As with the writing analyses, a sampling goal was to achieve a balance between 
students of UCLA Wriung Project Fellows and students of teachers having no exposure to 
the Project. While data coding and analysis procedures took all possible levels of teacher 
Project participation or exposure into account, analyses, in fact, demonstrated that teachers 
participating in this study eitner were Writing Project Fellows or had never received 
information or other exposure to the Project. Seven of these teachers reported being 
Writing Project Fellows and eight reported having no Project exposure. Hence, the teacher 
exposure/participation data were treated dichotomously. 



Results 



Writing Analyses 

T-tests of independent means and one way Analysis of Variance (ANOV A) were 
used respecUvely to determine the extent to which the quality of students' wriung is 
affected by (a) the level of CA1P training possessed by the students' current English teacher 
(i.e., level of teachw involvement) and (b) how many of the students' English teachers 
(current teacher plus past three teachers) had been CAIP Fellows. Since the current 
teachers of all students either were CAIP Fellows (labeled "Extensive involvement" for 
analysis and reporting purposes) or reported that they either had no knowledge of the 
UCLA/CAIP Writing Project or that they had heard of it but never participated in any way 
(labeled "Little involvement" for reporting and analysis purposes), the level of teacher 
involvement variable was treated dichotomously. 

Students' essays were scored as described above in the Methods section and as 
illustrated in the Appendix. Each essay received five scores: Holistic, General Impression. 



ERIC 



9 



Writing Impact 
9 

-Content Quality (GICQ), Principles/Concepts-Number (PN), Prior Knowledge-Facts, 
Events, and Experiences (PK), and Proportion of Text Detail (TEXT). Except for the 
TEXT score, high scores reflect positive results; in the case of this study, which was based 
on the "Hunger" passage by Richard Wright, low TEXT scores were most desirable. 

Results indicate that students of CAJP Fellows obtained significantly more positive 
scores on all five scales (p < .001). These results are summarized in Table 1. 



Insert Table 1 about here 



Results further indicate that the more CAIP Fellows that students have had as 
English teachers, the more positive their writing scores on all five scales. Using the CAP- 
bascd holistic scoring procedure (see Appendix ), students having one or more than one 
UCLA/CAIP Writing Project Fellow as an English teacher, out of their most recent four 
English teachers, scored significantly higher than students having no history of such 
teaching (p < .05 based on Tukey HSDs). 

Except for the TEXT score, results based on the Content Assessment scoring rubric 
are similar to those yielded by the CAP-bascd holistic scoring scheme, that is, students 
taught by one or more UCLA/CAIP Writing Project Fellows (i.e., one or more than one 
"exposure") outperformed students having no such exposures (p < .05 1 ,*scd on Tukey 
HSDs). The TEXT scores showed a significant positive difference between those students 
having more than one exposure to a UCLA/CAIP Writing Project Fellow compared to 
those having no exposures (p < .05, based on Tukey HSDs). 

Because variables such as grade point averages (GPAs), school level (junior high 
versus senior high school), and socioeconomic status (SES) may also have significant 
influences on student achievement, writing scores were additionally submitted to multiple 
regression analysis. The predictor variables were (a) level of student exposure to 
UCLA/CAIP Writing Project Fellows (0-4 1 TCLA/CAIP Writing Project Fellows as 
English teachers), (b) level of current teacher's involvement in the UCLA/CAIP Writing 
Project (extensive versus little, as described above), (c) self-report GPA, (dj average 
English GPA based on past three English grades, (e) school level (junior versus senior 
high school), and (f) school SES (based on most recent code used by CAP). Each of the 
writing scores (holistic, GICQ, PN, PK, TEXT) functioned as a criterion variable. 

Using holistic scores as the criterion variable, the first variable to enter the equation 
was average English GPA (R=.36, R 2 =.13, p<.001) ~ the higher the GPA, the higher the 
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writing scores. The second variable to enter was teacher involvement (total R=.46, 
R 2 =.21, p<.001) ~. students of UCLA/CAIP Writing Project Fellows outperformed 
students of teachers with little or no Project knowledge or experience ("little knowledge"). 
The third variable to enter 'vas school level (total R=.5l, R 2 =,26, p<.001), with high 
school students outperforming junior high school students. 

For the GICQ, PN, and PK criterion variables, school level and school SES 
respectively were the first two predictors to enter. For the GICQ and PK criterion 
variables, teacher involvement was the third predictor to enter; the third predictor to enter 
the equation for PN was average English GPA. A fourth predictor entered the equation for 
GICQ (average English GPA). The only significant predictor of TEXT was Teacher 
Involvement, with the relationship being negative as desired. A summary of these findings 
appears in Table 2; a correlation matrix is provided in Table 3. 



Insert Tables 2 and 3 about here 



Questionnaire Data 

Multiple regression analyses were used to determine the relationships among 
variables. Separate analyses were performed for each of 13 criterion variables: students' 
plans following high school (fulltimc continuing education versus part-time continuing 
education or fulltime work); type of continuing education institution student plans to attend, 
if applicable; type of institution to which SAT scores were sent, if applicable; and 
responses to items 16-25 on the questionnaire which tapped attitudes and perceptions about 
writing. Criterion variables for each of these analyses were: school SES, using the most 
recent code provided by CAP; the current English teacher's level of training and 
participation in the Writing Project (Fellow versus no exposure/participation); ethnicity; 
language spoken at home; level of students' exposure to Writing Project Fellows (ranging 
from one to four teachers over the last four English classes), self-report GPA; average 
GPA in high school English classes; and self-report score on the verbal portion of the SAT, 
if applicable. 

Using postsecondary plans for continuing education following high school as the 
criterion variable, the first predictor variable to enter the equation was GPA (R=.34, 
R 2 = 12, p<.001); the higher the GPA, the more likely students were to anticipate fulltime 
postsecondary education. The only other variable with significant contribution to the 
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prediction equation was ethnicity (R=-.14, R 2 =.02, p<.0i), with non-minority students 
being more likely to have fulltime postsecondary education plans th ;n minorities (blacks 
and Hispanics). 

When the criterion variable was type of postsecondary institution that students 
anticipated attending, the only significant variables to enter the equation were GPA (R=.48, 
R 2 =.34, p<.001) and average English GPA (R=.53, R 2 =.28, p<.(X)l). The criterion 
variable was assigned the following values: 4= a branch of the University of California, 
including UCLA; 3= a branch of the California State University system; 2= a California 
Community College; 1= a trade school. While the questionnaires allowed for other 
responses, there were very fevv; most such responses referred to the military or to private 
institutions. 

Multiple regression analysis based on only those students reporting the submission 
of SAT(Verbal) scores to a particular institution (n=133) showed average English GPA as 
the first predictor (R= 58, R 2 =.34, p<.(K)l) and overall GPA as the second (R=.61, 
R 2 =.37, p<.(X)l). For this analysis, the criterion variable was defined as the student's first 
choice school and was assigned the following values: 4= UCLA (n=28), 3= a branch of 
UC other than UCLA (n=35), 2= a branch of the California wState University system 
(n=52), and 1= a California Community College 01=18). Additional analyses revealed that 
78% of the Asian students responding to this item named UCLA or another branch of the 
University of California as their first choice, and no Asian student responding to this item 
listed a community college. However, only 17% of black students and 37% of Hispanic 
students,rcsponding to this item named UCLA or another branch of the UC system as their 
first choice; their first choices tended to be branches of California State University or a 
California Community College. 

Multiple regression findings related to items 16-25 on the Senior English Student 
Questionnaire (available from the author) are summarized below by item. Item means are 
provided in Table 4. 



Insert Table 4 about here 



9 

ERLC 



Item #16: I enjoy writing. The four significant predictors, in order, were ethnicity 
(R= 17, R 2 =.03, p<.01), with minority students enjoying writing more than non- 
minorities; average English GPA (R= 24, R 2 =.06, p<.001); GPA (R=.29, R 2 = 09, 
p<.001); and language spoken at home (R=.32, R 2 =.10, p<.001), with students from 
non-English speaking homes enjoying writing more than native English speakers. 

12 
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Item #1 7: / have always enjoyed writing. The two significant predictors were 
ethnicity (R=.20, R 2 =.04, p<.001), with Asian students reporting the highest levels of 
enjoyment, and average English GPA (R=.25, R 2 =.06, p<.001). 

Item #18: I now enjoy writing more than I used to. The two significant predictors 
were ethnicity (R=.14, R 2 =.02, p<.05), with blacks reporting the highest levels of 
increased enjoyment, and average English GPA (R=.18, R 2 =.03, p<.01). 

Item #19: I think lama good writer. The three significant predictors, in order, 
were language spoken at home (R=.22, R 2 =.05, p<.01), with native Spanish speakers 
reporting the highest levels of agreement with the statement; average English GPA (R=.26, 
R 2 =.07, p<.001); and ethnicity (R=.29, R 2 =.09, p<.001), with Asian students reporting 
the highest levels of agreement. 

Item #20: Others think that I write well. The three significant predictors, in order, 
were ethnicity (R=.21, R 2 =.04,p<.001), with black students reporting the highest levels of 
agreement with the statement; average English GPA (R=.27, R 2 =.07, p<.001); and 
language spoken at home (R=.30, R 2 =.09, p<.001), with native Spanish speakers 
reporting the highest levels of agreement. 

Item #21: Writing well is important to my future success. The three significant 
predictors, in order, were language spoken at home (R=.19, R 2 =.04, p<.00), with native 
English speakers reporting the highest levels of agreement with the statement; GPA 
(R=.23, R 2 =.06, p<.001); and ethnicity (R=.26, R 2 =.07, p<.001), with black students 
reporting the highest levels of agreement. 

Item #22: My writing skills seem to help me do better in other classes. The two 
significant predictors were average English GPA (R=.14, R 2 =.02, p<.05) and ethnicity 
(R=.19, R 2 =.03, p<-01), with Caucasians reporting the highest levels of agreement. 

Item #23: I plan on a job/career that requires good writing skills. The two 
significant predictors were GPA (R=. 16, R 2 =.02, p<.01) and ethnicity (R=.20, R 2 =.04, 
p<.01), with Asian students reporting the highest levels of agreement with the statement. 
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Item #24: After high school, I plan to write for pleasure. The only significant 
predictor was ethnicity (R=.14, R 2 =.02, p<.01), with black students reporting the highest 
levels of agreement with the statement. 

Item #25: / would like to write for a living. The two significant predictors were 
ethnicity (R=.23, R 2 =.05, p<.001), with black students reporting the highest levels of 
agreement with the statement, and average English GPA (R=.28, R 2 =.08, p<.001). 

Conclusions and Recommendations 

While level of teacher involvement in the Writing Project and level of student 
exposure to Writing Project Fellows were significant predictors of writing quality, not 
surprisingly, in most instances, other variables were of greater significance in their 
predictive power: GPA was the strongest predictor of holistic writing scores and school 
level (high school versus junior high school) and school SES were stronger predictors of 
General Information and Content Quality (GICQ), Principles and Concepts (PN), and Prior 
Knowledge (PK) scores. Nonetheless, the results of this study make it quite clear that 
students in the classes of teachers who are Writing Project Fellows write significantly better 
than students who are not. It is also clear that students' writing improves in direct 
proportion to the number of Writing Project Fellows they experience as English teachers. 

Findings related to the effects of Writing Project involvement on students' GPAs, 
postsecondary plans, test scores, and attitudes toward writing are less clear. The available 
data, which are limited by their self-report nature, suggest that GPAs and ethnicity are most 
predictive of students' decisions about whether and where they will continue their 
education after high school. 

While this study was limited by available resources and the context in which it was 
conducted, it provides useful findings and important insights for future evaluation efforts. 
Aspects of the present approach that should be retained include the systematic collection 
and analysis of student products such as writing samples and design consideration of 
moderator variables such as language spoken at home, SES, school context, and intensity 
and validity of the intervention under evaluation (i.e., student exposure to Writing Project 
Fellows). The present approach could be improved by collecting writing or other work 
samples and self-report data from some of the same students, by depending less on self- 
report data (e.g., grades and test scores), and by obtaining student-level rather than school- 
level SES information. 
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The success of future efforts to assess the impact of outreach programs on students' 
decisions to enter the University system and their university success requires careful 
attention and commitment from central administration to developing mechanisms for 
tracking students in at least two ways: (a) tracking cohorts of current university students 
backward to determine the nature and extent of their involvement in targeted outreach 
programs and (b) tracking targeted high school students forward (longitudinally) to 
determine how various levels of involvement in outreach programs affect their future 
planning, decision-making, and success. 
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